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Background / Context: 

Description of prior research and its intellectual context. 


Recent years have seen an increased interest in quantitative educational research studies that use 
random assignment (RA) to evaluate the causal impacts of educational interventions (Angrist, 
2004). The multi-level structure of the public education system in the United States often leads 
to experimental designs where naturally occurring clusters (eg. schools) are utilized to recruit 
participants (eg. students) into a study and/or as units that are randomized into one of two 
experimental conditions. 

When multiple clusters are recruited to participate in a research study but individuals within 
clusters are randomly assigned to experimental conditions the resulting experimental design is 
often referred to as a multi-site design. When clusters are utilized as the unit of randomization 
the resulting experimental design is generally called a cluster randomized or hierarchical design. 
While designs that use some form of random assignment are generally preferred for making 
causal inferences about treatment effects, RA designs are not always practical or feasible 
(Shadish, Cook and Campbell, 2002). The next best alternative to a random assignment design is 
often the regression discontinuity (RD) design. Studies using the regression discontinuity design 
can be constructed when assignment to treatment occurs as a function of a fully observed score 
variable which has a continuous distribution. Units with scores below a preset cutoff value are 
assigned to the treatment group, and units with scores above the cutoff value are assigned to the 
comparison group, or vice versa. 

The same clustering issues that arise with random assignment studies also impact regression 
discontinuity studies. The score variable may be measured at any of many different levels in the 
educational hierarchy while outcome measurements are obtained at a lower level in the 
hierarchy. These sorts of situations lead to hierarchical regression discontinuity (RD) designs. 
Alternatively, the units chosen to participate in the study may be clustered in certain sites, even 
though the score variable is measured within sites. The resulting study is a multi-site regression 
discontinuity design. 

An important component of planning experiments with clustering (whether they be multi-site or 
cluster randomized designs) is to determine the optimal within and between cluster sample sizes 
subject to a cost constraint. The optimal sample size at each level of the design has been reported 
for two level cluster randomized designs (Raudenbush, 1997), for two level multi-site 
randomized designs (Raudenbush and Liu, 2000) and for three level cluster randomized designs 
(Konstantopoulos, 2009). However, no solution has been provided for the problem of optimal 
design in regression discontinuity studies. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


The current paper builds on previous work by Schochet (2008, 2009) in order to provide 
formulas for the optimal sample size at each level of a two-level regression discontinuity design. 
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The paper also provides optimal sample size formulas for unbalanced random assignment 
designs (such formulas have not previously appeared in the literature). Optimal sample sizes for 
RD designs are compared to the optimal samples sizes for the corresponding RA designs. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 


Recent work has indicated a resurgence of interest in using regression discontinuity designs for 
estimating the causal impacts of educational interventions (Cook, 2008; Imbens and Lemieux, 

2008) . Previous work has provided formulas for computing the variance of impact estimates 
based on regression discontinuity designs (Schochet, 2009). There has also been work 
identifying for researchers the optimal sample sizes to use at different levels of the hierarchy in 
random assignment designs (Raudenbush, 1997; Raudenbush and Liu, 2000; Konstantopoulos, 

2009) . However, there has not yet been published work illustrating optimal sample sizes for 
regression discontinuity designs with a hierarchical structure. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 


We assume that the study in question contains n clusters, each consisting of m individuals who 
participate in the study. Let Y\ s represent the outcome measured on the jth individual in the ith 
cluster. In order to ensure comparability of results from RA and RD designs, we assume that the 
score variable is also measured and included in statistical models for RA designs. 

We assume that the researcher has a fixed budget constraint labeled T. The marginal cost of 
including an additional cluster in the study is C 2 , and the marginal cost of including an additional 
individual from a cluster that has already been sampled is C/. We assume the following simple 
linear cost function defines the cost constraint: 

T = C 2 n + C 1 mn. (1) 


Hierarchical random assignment and regression discontinuity designs. 

The standard linear mixed model used to model hierarchical designs can be written as follows: 

Yy = a 0 + (X ] T j + a 2 ( Score c - Score ,, ) + afScore + r t + s tj . (2) 


Tj is a treatment indicator variable coded -0.5 for control cases and 0.5 for treatment cases, and 
we assume that a proportion, P, of the available clusters is randomized to treatment. Score t j is the 
measured value of the score variable for individual j in cluster i. Scores is the average value of 
the score variable in cluster i. The v, random variables are random cluster effects and have 
variance cr\ T and the sq random variables are random individual level residuals having 


variance o\ . The “A” subscripts are included to emphasize that the variance components are 
“adjusted” in the sense that the variances are computed after conditioning on the score variable. 
The (adjusted) intracluster correlation coefficient (ICC), pA, is defined as 


cr 


Pa ~ 


A,t 


(3) 


cr 


A,T 
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Here cr 2 AT = a\ r + <j~ a is the total (adjusted) variance in the outcome variable of interest. 


Multi-site random assignment and regression discontinuity designs 

The regression form of the standard linear mixed model used to model multi-site random 

assignment designs can be written as follows: 

Yy = a 0 + aj] j + a 2 (Score {j - Score ,, ) + a, Score, , + j3 i + (aft), 71. + . (4) 


In equation (4) the ff random variables represent variability in the level of the outcome across 
clusters and have variance p . The af3\ random variables represent variability in the treatment 


effects across clusters and have variance <J 2 Aap . The “A” subscripts are again included to indicate 


that the variances are conditional on the score variable. Other quantities are defined as they were 
in equation (2) with the exception that we now assume a proportion, P, of units within each 
cluster are randomly assigned to treatment. In order to ensure that the total variance of the 
outcome variable is the same in the multi-site and cluster randomization formulations of the 
model it is necessary to make the following equivalence between the variance component 
parameters 


, — cr~ „ H — cr 


A,T 




A,a/3 * 


( 5 ) 


It may be useful to think about the variance in treatment effects relative to the total 
variance across clusters. As such, we define the parameter coa as follows: 


( °a=- 


A.a/3 / 


(6) 


A,t 


Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

The usefulness of the results is evident from the description in the Findings/Results section. 

Findings / Results: 

Description of the main findings with specific details. 


We use the subscripts “H”, “MS”, “RA” and “RD” to represent “Hierarchical”, “Multi-site”, 
“Random assignment” and “Regression discontinuity”, respectively. So, for instance, a result for 
a multi-site regression discontinuity design would be subscripted “MS, RD”. With this 
notational convention we obtain the following results. 


The variance of the treatment effect in a hierarchical RA design is given by the following 
equation (Bloom, 2005): 

1 1 


Var HRA (a i) = cr 


A,T 


-(] + (m-\)p A ) . 


P(l-P) mn 

When a hierarchical RD design is considered, the equation is modified to read (Schochet, 2008): 

2 1 11 

Var H R D («i ) = cr AT — — (1 + (m - 1 )p A ) . 

(1 -p TS ) P(}~P) mn 


( 7 ) 

( 8 ) 
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The notation “ p\ s ” represents the correlation between the score variable and the treatment 
indicator variable. 


Using the method of Lagrange multipliers to optimize equation (7) subject to the constraint given 
in (2) yields an optimal within-cluster sample size of: 


m opt,H,RA 


EE IE 

Pa V C i 


(9) 


The constrained optimization of equation (8) yields the same result as the constrained 
optimization of equation (7). This leads us to our first major finding: the optimal within cluster 
sample size for hierarchical RD designs is that same as the optimal within cluster sample size for 
hierarchical RA designs. Furthermore, the optimal within cluster sample size does not depend on 
the proportion of clusters randomized to treatment. 


The variance of the treatment effect estimator in multisite RA designs is given by the following 
equation (Hedges and Rhoads, 2009): 


Var MS.RA( a 0 = 


CJ 


A,a/3 


1 


cr: 


MS ,RA s ' D/1 n\ 

n P{ 1 — P) mn 

It is straightforward to derive the corresponding results for multi-site RD designs: 


Var, 


MS,RD 


{a i) = 


CJ 


A,ap 


1 


1 


cr. 


P(l-P) (1 - Pts ) nm 


Constrained optimization of (10) yields: 


m 


opt, MS ,RA 


1 j 

I 1 ~Pa I 

pP(l-P)]l 

(0 aPa V 


And constrained optimization of (11) yields: 


1 


m 


\ 1 ~Pa 


opt, MS ,RD= 


1 2P(1 - P)(l - pj S ) y co A p A 

In contrast to hierarchical designs, in multi-site designs the optimal within cluster sample size 
will depend on both the proportion of units randomized to treatment and on the extent of the 
correlation between the treatment indicator variable and the score variable in RD designs. 
Designs that are highly unbalanced will result in larger within cluster sample sizes, as will 
designs where the value of Pj S is large. A sampling of the optimal within cluster sample sizes 
obtained under the various design scenarios considered is presented in Table 1 in the appendix. 

C 

Results are presented for various cost ratios (values of — ), proportions in treatment (P), values 

Q 


( 10 ) 


(ii) 


( 12 ) 


(13) 


of the conditional ICC ( p A ) and values of the treatment effect heterogeneity parameter (co A ). 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


This study has shown that the optimal within cluster sample size for a two level hierarchical 
design is the same regardless of whether the study is a regression discontinuity study or a random 
assignment study. The optimal sample size in hierarchical designs is also insensitive to 
differences between the number of clusters in the two conditions being compared. 
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The story is different when multi-site designs are considered. Regression discontinuity studies 
should be structured to have more units within each site than random assignment designs. 
Additionally, if the design is unbalanced (more units in one condition than the other) the within 
cluster sample size should be increased. 
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Appendix B. Tables and Figures 

Table 1: Optimal within cluster sample sizes. 
Cost ratio ICC P co 

Hier opt. m 

MS opt. m (RA) 

MS opt. m (RD) 

2 

0.05 

0.5 

0.05 

6 

39 

65 

2 

0.05 

0.5 

0.5 

6 

12 

21 

2 

0.05 

0.3 (or 0.7) 

0.05 

6 

43 

66 

2 

0.05 

0.3 (or 0.7) 

0.5 

6 

14 

21 

2 

0.05 

0.1 (or 0.9) 

0.05 

6 

65 

80 

2 

0.05 

0.1 (or 0.9) 

0.5 

6 

21 

25 

2 

0.15 

0.5 

0.05 

3 

21 

36 

2 

0.15 

0.5 

0.5 

3 

7 

11 

2 

0.15 

0.3 (or 0.7) 

0.05 

3 

23 

36 

2 

0.15 

0.3 (or 0.7) 

0.5 

3 

7 

11 

2 

0.15 

0.1 (or 0.9) 

0.05 

3 

36 

44 

2 

0.15 

0.1 (or 0.9) 

0.5 

3 

11 

14 

2 

0.25 

0.5 

0.05 

2 

16 

26 

2 

0.25 

0.5 

0.5 

2 

5 

8 

2 

0.25 

0.3 (or 0.7) 

0.05 

2 

17 

26 

2 

0.25 

0.3 (or 0.7) 

0.5 

2 

5 

8 

2 

0.25 

0.1 (or 0.9) 

0.05 

2 

26 

32 

2 

0.25 

0.1 (or 0.9) 

0.5 

2 

8 

10 

5 

0.05 

0.5 

0.05 

10 

62 

103 

5 

0.05 

0.5 

0.5 

10 

20 

33 

5 

0.05 

0.3 (or 0.7) 

0.05 

10 

67 

104 

5 

0.05 

0.3 (or 0.7) 

0.5 

10 

21 

33 

5 

0.05 

0.1 (or 0.9) 

0.05 

10 

103 

126 

5 

0.05 

0.1 (or 0.9) 

0.5 

10 

33 

40 

5 

0.15 

0.5 

0.05 

5 

34 

56 

5 

0.15 

0.5 

0.5 

5 

11 

18 

5 

0.15 

0.3 (or 0.7) 

0.05 

5 

37 

57 

5 

0.15 

0.3 (or 0.7) 

0.5 

5 

12 

18 

5 

0.15 

0.1 (or 0.9) 

0.05 

5 

56 

69 

5 

0.15 

0.1 (or 0.9) 

0.5 

5 

18 

22 

5 

0.25 

0.5 

0.05 

4 

25 

41 

5 

0.25 

0.5 

0.5 

4 

8 

13 

5 

0.25 

0.3 (or 0.7) 

0.05 

4 

27 

41 

5 

0.25 

0.3 (or 0.7) 

0.5 

4 

9 

13 

5 

0.25 

0.1 (or 0.9) 

0.05 

4 

41 

50 

5 

0.25 

0.1 (or 0.9) 

0.5 

4 

13 

16 

15 

0.05 

0.5 

0.05 

17 

107 

178 

15 

0.05 

0.5 

0.5 

17 

34 

56 

15 

0.05 

0.3 (or 0.7) 

0.05 

17 

117 

179 

15 

0.05 

0.3 (or 0.7) 

0.5 

17 

37 

57 

15 

0.05 

0.1 (or 0.9) 

0.05 

17 

178 

218 

15 

0.05 

0.1 (or 0.9) 

0.5 

17 

56 

69 

15 

0.15 

0.5 

0.05 

9 

58 

97 

15 

0.15 

0.5 

0.5 

9 

18 

31 

15 

0.15 

0.3 (or 0.7) 

0.05 

9 

64 

98 

15 

0.15 

0.3 (or 0.7) 

0.5 

9 

20 

31 

15 

0.15 

0.1 (or 0.9) 

0.05 

9 

97 

119 

15 

0.15 

0.1 (or 0.9) 

0.5 

9 

31 

38 

15 

0.25 

0.5 

0.05 

7 

42 

71 

15 

0.25 

0.5 

0.5 

7 

13 

22 

15 

0.25 

0.3 (or 0.7) 

0.05 

7 

46 

71 

15 

0.25 

0.3 (or 0.7) 

0.5 

7 

15 

23 

15 

0.25 

0.1 (or 0.9) 

0.05 

7 

71 

87 

15 

0.25 

0.1 (or 0.9) 

0.5 

7 

22 

27 


Note: Table assumes a normally distributed score variable 


SREE Spring 2014 Conference Abstract Template 


B- 



SREE Spring 2014 Conference Abstract Template 


B-2 



